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ABSTRACT _ t 

A nummary analysis Is made of problems encountered by retrieval staffs 1 n three 
states during the initial year of the Pilot State Project in Information. 
Dissemination, Problems discussed are; (l) Computer Related; (2) Furnishing the 
client with complete copy; (3) Record keeping and Filing; and (4) Staffing, 
Recommendations resulting from the experiences of the first three p i 1 ot . states , 
and which might be useful to other states that are attempting to establish 
information dissemination projects, are: (l) A state Lhould plan initially to ^ =* 
provide its own computerized l, ear eh capacity; (2) A state that decides^ to instal i 
QUERY might well plan to have their computer and retrieval personnel visit a 
similar installation that has already been through the process, so that ways of 
overcoming problems can be learned; (3) The Office of Education or the QUERY . 
contractor should provide assistance to new Installations until the program is 
operational; (4) The rationale behind packaged information should be well 
understood; (5) Retrieval services should adopt a means of continual reassessment 
of their methods; (6) Procedures for completing the process should be determined 
in advance; (7) Personnel of an information service would do well to visit an 
^existing retrieval service and analyze their record-keeping and filing systems; _ 
and (8) The qua 1 i f i cat lens and capabilities of the staff should be determined 

in advance, ( DB ) 
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PROBLEMS IE- lEFORMATrOE RETRIEVAL 



After the initial year of the Pilot State project in information 
dissemination, some summary analysis of the problems encountered by 
the retrieval staffs in the three States might usefully be attempted. 
The energy, ingenuity and unflagging effort with which retrieval 

personnel have met these problems and attempted to work out solutions 

> 

have b.een commendable; but, in general, these issues have been en fa 

\ 

countered and dealt ..with on an ad hoc basis* Understandably enough, 
what would be the basic and most problematic issues to be cun,, anted 
in setting up a central State educational information service could 
only be partially foreseen a year ago by any State, and not very 
clearly outlined* This, of course, is the purpose of pilot projects — 
“§o that such issues can be more carefully delineated in advance, and 
problems can be anticipated and perhaps avoided or alleviated for 
similar undertakings later on# ' ‘ ’ 
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I* Computer Related Problems ^ 

The first group of problems relates to the objective of establishing 
a computerized information retrieval capability for the information 
service in each State, the area which has presented the most 
time-consuming and expensive difficulties* 

A. Turnaround time 

The two western States in the project depended on a large regional 
computerized information retrieval center during their initial year 
of operation. Both States have expressed satisfaction with their 
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relationship with the Regional Center and have commented favorably 
—©a its various services. State A, in particular, has emphasized that 
they have been relatively pleased with the Regional Center, but 
observation of this arrangement has led the evaluation team to the 
judgment that not having control over all the steps involved in the 
computerized retrieval process created some special problems for 
these Jtwo States. 

One of the problems evident in the Regional Center service has 

i 

\ 

been excessive turn-around time. The project director of State A 
anticipated this difficulty very early. He sent two test requests 
for information to the Regional Center before his State *s project was 
officially underway and did not get a return for thro a weeks. This 
led State A in the earliest phases of its project to provide computerized 
—searches within their own State (although they are still n ot o pe ratio nal 
tn terms of economic feasibility^^ as of September , 1971) * That State's 
later experiences with the Regional Center supplied continuing * . ■ -- 

justification for this decision. In the early weeks of the project* 
their average turn-around time on requests for information referred 
to the Regional Center for a search of ERIC was 4,7 weeks. This was 

#> i*. = 

figured on the basis of dates on the office log kept in State A for * 
requests received in October and the first two weeks in November, 

There was some improvement in early December* -turn-around time for 
~ one request was two weeks, and for another under three weeks --but it 
was not consistent. No request sent to the Regional Center on or 
after Friday, December 11, 1970, had been returned by the end of the 
year* In mid- January , the Center closed down altogether* 
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The lengthy turn-around was true only for requests referred to the 



Regional Center* . Requests not sent there but handled in the State A 



Re trieval4»D is semination Center during those particular weeks, when the volume 
of requests was still relatively low, received literally same-day service— or, 
for a request received on Friday', information would be returned on Monday® 



The same story held true in State B • Average turn-around time for I 

- . I 

requests referred to the Regional Center (figured on the basis of dates on 

> 

the office log sheet for requests received in October) was almost four and 
a half weeks. Again, requests handled in the Sta e Information Center, and 
not sent to the Regional Center, were answered quickly. Also, the State 
Information Center occasionally would send information which it could collect 
or had on hand to the requester, and then follow this up with the computer 
printout or profile of abstracts of documents available in ERIC files on that 
subject when it finally arrived. 



Thus, in both western States, requests which could be answered with 
‘materials available in the information center, or through manual searches 

done In the State library or elsewhere, or by other personal and local 

■. , \ ^ . 

efforts of the retrieval staff were serviced more speedily than those answered 

• , \ 

through computerised searches. But although these locally handled requests 
received faster service, they were not, of course, answered on the basis of 
an extensive survey of the vast national pool of educational research, as 



was intended to be done through computer searches. It is unrealistic to expect 
reference center staff members to do manual searches of ERIC indexes once the 



volume of requests reaches a high level* 

These figures for States A and B can be contrasted with an average 
turn-around time of two and a half weeks for State C in the early stages of 
the project, (State C*s figure came from dates on a log of 43 requests received 
in December , However, the average turn-around time was figured only on the 



basis of the 20 of those 43 requests which had been completed — 

of the remaining requests, 14 were "in process" and 9 were "pending". 

Thus* the figure may not be exactly comparable with those for States A and B, 
where the averages were based on a much higher proportion of the total load 
of cases, but at least some of the State G requesters were receiving reasonably 
fast service, including computerised searches,) Coding limitations of the 
computer program and excessive computer time to run the program made request 
processing a slow procedure- Most probably, the same problem had developed 
in the Regional Center- States A and B simply forwarded the written request 
forms to the Regional Center , and coding for the computer search was done 
by its staff. Possibly the delays were due to an overloaded staff doing chis 
work, not to an overload or a backlog for the computer itself. 

The Regional Center shut down its service comple tely January 14, 1971 
for reprogramming* The retrieval center director in State A recalls that 
there was approximately a month's advance notice that the service would bev . ^ 
Inoperative for a while (they were closed about three weeks) so that the 
field agents - and through them potential users - could be warned ahead of 
time# State A continued sending requests to the Regional Center during that 
period, allowing the backlog to accumulate there. Meanwhile, the staff in 
State A did manual searches when possible to answer requests; but the necessity 
of a shutdown contributed to even lengthier delays in replying to users. In 
State 1, dissatisfactions with the Regional Center and turn-around time had 
already prompted the project staff to provide some alternative methods of 
answering request®, but resources and supplies for these methods had not yet 
been received. Thus, during the period of the shutdown, for the majority of 
the cases, requests for information just could not be filled. Thus, only 
three of the 63 requests for ERIC searches in January ware filled that month. 



Since that time the Regional Center has modified its method of opei-ition. 

A great many of the requests for information referred to them now are answered 
with ’'packages' 1 * that is* packets of research abstracts and other materials 

I 

on a previously prepared listing of topics. The main reason for the new ; 

packaged services was the length of time required for the Search in Depth, 
or BID, as the separate computer search for an individual request is termed. 

i 

SID’s are still performed to answer a request for information from the project 

s 

States If the Regional Center decides it is necessary, which is usually j 

based on a decision that none of the packaged services is applicable or adequate.’ 



State A retrieval personnel report that the average turn-around is ^ ill 
three weeks if an SID is done on a request. A one-week turn-around was 
promised by the Regional Center for its new packaged services , and both 
States 'A and B repor ted this summer (1971 ) that .it , is meeting that promise. 
The State A director estimated that the total turn-around to a user for the 
pre-packaged material in answer to a, request would average about nine days. 



including mailing time for receipt of the request and return of the material 
•by the State Information center, *"V * 

It should be noted* however, that this retrieval center director believed 
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that the concern about lengthy turn-around time may not be matched by 
irritation from the users. Few clients in State A have complained or said 
the service was too slow, even when questioned specifically about this* 

B* Irrelevancy and inadequacy of materials 

— — . — - ^r 

- There have also been concerns about the quality of the output* a matter 
which must inevitably be considered with any computerized retrieval system. 
States A and B could try to assess the output received, but because they 
were relying on a distant computer staff, they were hampered in efforts to 
influence the methods used in the process or to improve the product* 
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The bulk of eke returns from the Regional Center has consisted of 
computet printouts of abstracts (of articles, reports or other documents), 
supposedly all concerning the subject on which the requester wanted information, 
These printouts result from a computer search of all the data and research 
in ERIC, Althou^i theoretically the regional installation could draw on a 
wider range of resources, any of these - even the CIJE (Current Index to 

i 

Journals in Education) files - required manual searches, and the Center was 
not required to supply manual searches to the two pilot States (although 
it did to its other clients) 7*' Output from the wider range of sources is 

included in one of the~ packaged services which the Regional Center is now 

\ 

offering. There have been occasional instances when copies of other documents 
were received but, in general, returns to the Pilot States have been restricted 
to a profile (or printout of abstracts) of ERIC documents. The relevance 

s.. . — — 

and adequacy of the output have both caused concern. State B throughout the 

\ 

first months of .he project voiced repeated criticism about the high percentage 
of the listed articles or documents which seemed obviously irrelevant to 
the client's interest, while in State A. the paucity of research turned up 
on particular topics was frequently noted. 

As yet we have no systematic check of the users' evaluation of the in- 
formation returned to them. State B recently received a compilation of figures 
based on. the Regional Center 1 s evaluation forms, but the response -rate was 
so poor that these statistics cannot be relied on for definitive conclusions. 

If one can assume, however, that the forms were more often returned by clients 
who were satisfied with the service (which is usual in this type of survey), 
then it is noteworthy that 36 percent of these 109 clients indicated that 
less than half of the computer searches (abstract^ were pert in ent to their 
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request. Incidentally, one of our observers mentioned at a staff meeting in 
that State in December that he had watched a field agent glancing over a 
P ro ^^ e before returning it to a client and immediately marking off at least 
75 percent of the items as obviously irrelevant. 

The State A retrieval director estimates that there are no relevant 
abstracts at ail included in the listing for only about 10 percent of the 
cases returned' by the Regional Center. "There’s almost always something 
that’s useful,' at least one^artiele , " he said. His impression is that a 
tabulation of clie nt res ponses on the form included in the Regional Center 

packet would show 45 to 50 percent of the users assessing about half of the 

\ 

output as relevant* 

In the beginning, this retrieval director was quite concerned about 
several instances- in* which it seemed that there was no- research available in 
ERIC on a topic,- After a -random examination of cases in the "closed" file, 
our observer in that State cited several specific instances where the ERIC 
search produced little useful material or very few articles, or where there 

seemed to be nothing at alUn ERIC directly on the subject. A request, for 

\ 

reports on ungraded social studies programs for junior high schools, for 
example, elicited only four articles. Another requester asked for information 
on two different topics — computer programs for assisting instruction and 
performance-contracting as related to intelligence testing; ERIC produced 
nothing on either topic (although the retrieval center attempted to help the 
requester through other means). For a request for material on district-wide 
coordination in rural areas, the return from the Regional Center included 
nothing on curriculum coordination, which was the main point of the requests 

The retrieval director's concern about this problem has been somewhat 

* 

alleviated because he has not encountered the irritation from users which he 

s. ' • 



anticipated * 'The clients are not as disappointed on that point as we (the 

retrieval staff) are. They are just as interested sometimes to learn that 

there is no research on their topic he- re ported . 

With the topics on which research is actually conducted being determined 

by such a diverse and haphazard collection of factors, not the least influential 

being the individual curiosity of investigators, the explanation for the inade- 

« 

quacy of these responses to requests may well be that there are surprising gaps 
in the educational research literature - that there literally Is no research on 
that topic. Or it may be tha? pertinent research on topics gets lost or is 
not found by any specific computer search because of faults in the indexing or 
categorization of materials put into the data bank, or on the computer tapes. 
Personnel ir a state retrieval office would have little remedial power if the 
explanation is either of these factors, other than possible long-range effects 

■Sw. Mg j 

through spotting examples of such omissions or defects and making Office of 
Education .officials or others aware of then. 

But the explanations for disappointing output from computerized retrieval 
of information may be on a lower level# ^ Some other factors which influence the 
success or failure of such an operation are: 

1) The wording of the topic cn which information is requested* The 
requester may have been rather uncertain about what his question was or he may 
not have articulated it specifically enough even if he had a very clear idea; 
and the field agent may not have spotted this inadequacy or made up for it* If 
the retrieval center in the state is ordering the computer search <i.e#, coding 
the information request for the computer ) p this problem is more likely to be 
recognized and to be remedied inasmuch as can call the field agent and get him 
to recheck with the user on exactly what he wajits. If the retrieval center is 
simply relaying requests to a far-off service, the wording is probably scrutinized 
"■ess carefully and deficiencies or ambiguities are less likely to be noted* 



9 



2) The instructions or coding of the order for the computer search. 
The coder may misinterpret the intent or meaning of the request or may 
not use the most productive terms in ordering the search. For example, 
one request was for information on the rationale for parent- teacher 
conferences. The Regional Center used the descriptor "school- community 

relations" and returned a listing of articles which were mostly irrelevant 

s 

to the request. The State reference center directory experimenting 
with their new computerized -retrieval capability used the term "parent- 
teacher conferences^- -and received in return a listing of twenty articles, 
all of which looked relevant. A State retrieval office could make similar 
errors in ordering their own computer searches, but at least they would 
have some recourse — mistakes might be spotted and a search redone. In 
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an instance like this, if the State staff had not been able to experiment 
with different descriptors, they might have assumed that the fault was 
not with the coding of the request but that there was, in fact, no 
research available on that topic. 

In State Q where tl^e project uses its own computer, retrieval 

personnel responsible for ordering computer searches point out that relevancy 
the output Is almost entirely a result of the coding: relevancy may vary 

from 2 percent to 100 percent depending on deliberate decisions made by the 
coder in ordering the search. If the retrieval expert suspects there will be 
little research on the topic of a particular request , instructions to the computer 
may be written so as to extract every bit of data that might even conceivably 
bear on the subject, with full awareness ahead of time that in so doing a good 
many useless and Irrelevant listings will be cited as well. In another case, 
the staff member might be aware that a great deal of research had been done in 
the area of the topic and the aim in coding would be to limit the output in advance. 
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to instruct the computer to exclude much material and list only items fitting 
much more specific definitions. 

(3) Economic factors involved in computerised retrieval, especially 
a regional center doing big-volume business, and limitations of the program 
used on the computer. The relevancy of the computer search output especially 

| 

may be dependent on such factors. For example* it simply may not be economically 

\ _ _ _ ■ # ! 

feasible for a regional center, with big volume business and far-distant ] 

* ] 

clients, to do specifically tailor^ searches. They may find it frequently J 

necessary to settle for searches on a more generalized level. 

Whatever the explanation for the irrelevant and inadequate returns, 
states depending on a computerized search from a far-distant source are 
grossly handicapped in tacking these problems because they are unable to 
fflpp-ftor the entire retrieval process. To whatever extehtjmy of these 
three listed factors are involved (and most especially #2 or #3) , a state 
not control ling,? its "own computer searches has little hope of taking^ remedial 
action, 

— . — * 

G, Pre^oackaaed Information 

# * * . , 

In the early months of 1971 Regional Center administrators visited 

the retrieval offices in the two pilot states to explain and "sell" the new 
packaged services they were to start offering. Given the record to that 
point, both states might have been disinclined to continue the Regional 
Center service, but— primarily because of these new packaged services --both 
decided to continue, on a paying basis. The new services are: 
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Selected, materials on high interest topics, including a combination 
of ERIC abstracts and xeroxed articles from current journals. These packets 
provide "survey-type background information" on topics such as accountability, 
behavioral objectives and individualized instruction, topics which have been 
the subject of numerous requests for information received in recent years, 

A packet, of ERIC abstracts in a specific subject or grade level area: 
art education, early childhood education, elementary school counseling, and 
so forth. ~ 

A quarterly~'tevtew of the most recent ERIC abstracts in a particular 
area. There are 30 titles or categories so far defined and listed in the 
quarterly review. Once an individual is identified as interested in 
information in one of these areas or categories, he is automatic ally sent 
abstracts of the more recent ERIC reports in that area each quarter. Some, 
of the categories are educational facilities, educational finance , language 
arts, etc. In a way, the service is similar to subscribing to a specialized 

professional journal, ■ ^ / /. 

These services are essentially pre-packaged materials on various topics 



stacked on the shelf , "waiting for order ? 



in one retrieval director 1 s 



terminology,* State A attempted to buy only the packaged services— or to 
get one copy of every package developed which they might then duplicate 



themselves— but the Regional Center refused. Obviously, the regional service 
was hoping to subsidize its expensive and time-consuming individual computer 
searches with the new pre-packaged services. The Regional Center itself 
finally determines which requests are answered through individual searches 



ERIC 



♦Tti for ma tion received this summer indicates that as yet the contents of the 
packages are not definitely set. In August, the Regional Center was rerunning 
the computer search each time a request for one of the "packaged services 
was received, and hard copy of the complete articles to be included in any 
packet was duplicated after the specie request was on hand. • 
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and which by packages. In State A, approximately half the requests forwarded 

to the Regional Center are still receiving an SID. But information on the 

new servlces--on how to order them and on all the packaged categories— has 

been widely disseminated, both to field agents and to users in the two states, 

and many of the requests uow received in both states are for specific 
( 

packages, or on one of the topics in the listings. A client sometimes 
requests a specific package and also an SID on the topic in which he is 
interested. " 



Project personnel in both states consider the packages and the new 
services to be excellent. In early March, the State A retrieval director 
reported at a staff meeting that the regional service had a three-week 




turn-around time even on its packages (as it still h as fui ■ Sip -' s) a nd — 
concluded that they may have been "a little premature in their advertisement" 
of the new services. However, before the summer, both states reported that 
the Regional Center was meeting its promise of one-week service on the 
packages --fas ter service had been the most emphasized selling point of the 
modified basis of their service. 

Still, it should be noted that there are points of concern, about 
packaged information. Even though packages are rated very high on quality 

f 

of materials and research included about a topic, requests for information 
answered in this way do not receive quite as much individual service and 
"tailoring" as those answered by an SID. Another continuing concern will 



be how frequently the packages are updated. The Regional Center has promised 
that all the packaged services will be updated every two to three months, but 
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retrieval centers answering requests for information with packaged materials 
may need to check constantly to insure that its clients are receiving the 
most^ current research on these topics. Finally, and most important, clients 
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given a list of topics on which information is available are being approached 



in an entirely different way than clients who are urged to define their 



problems or needs for information, with promises of research information or 
other assistance which might be useful to them* To a large extent, jdjjl 
problem or need is defined by the range of available packages, rather than 



i 



by the individual client. To be sure, both in-depth searches and packages 



can be used at the same time. Also, a retrieval center doing its own 
searches is likely to attempt to develop its own package to answer an 



often- repeated request In. its locality, or at least to economize and stream- 



line its operations by not duplicating a recently done search. In any 
event, project staffs in the states are finding that the new packages are 

being 'well received by users and are frequently requested. 

* . * * . - " * 

D. Costs N 

■ v * ■ • * . * • 

There. is ticf completely valid basis for arriving. at cost-per- request 

figures for the Regional Center service, since the states did not pay for the 

service during the initial-year and presumably* the. cost for the across-the- 

board individual search service would have been higher than that set for the 

* * o. 

second year with much of the service on a packaged basis** But, with this 
disclaimer aforehand, we have attempted some elementary arithmetic to get a 
better sense of the cost of the computerized retrieval capacity to the two 
western project states* Using the volume of requests received in the first 






* One state reported a $15*00 charge and the other an $8*00 charge by the 
Regional Center for any clients not legitimately included in the project 
whose requests war' 1 nonetheless relayed by the State retrieval center 
for servicing by the Regional Center* We consider these figures to represent 

charges-per-requesti rather than accurate y assessed costs-per-request and 
thus have not used them to calculate what the Regional Center service would have 
cost the state during the initial year, if they had paid for it. The cost during 
second year, as reported to us, will.be $^,200 for State B and $4)500 for 
State A for each six month period. We have not explanation as yet for the 
discrepancy in costs for the service repo^^ to us b/ the two states. 







part of 1971 (the projects were not well underway befoi-e January, 1971, so 
including months previous to that would only distort the figure further) 
and the cost estimated for the Regional Center service in fiscal 1971-72, the 
cost would have averaged slightly more than $10*50 per request in both states ** 
It is possible that there will be no real ground for a comparison between 
the qos t-per-request during , the first year of the project and during the 
second, since the nature of the service is so drastically different. The 
new packages are being widely publicized, for example, in State B. The 
Project Director there estimated the cost per search at $8 (on the basis 
of charges from* the Regional Center during the previous year to paying 
— — . ■■■■■- 

** This is figured on ♦'he following basis: For State A, a total of 422 

requests for the six-month period January through June, 1971, divided into 
the cost of $4,500 set for the next six months, for an average of $10.65 
per request. This assumes that all State A requests were referred to the 
Retrieval Center but a small percentage were not* For State B, the 260 
requests received during the first four "* °7 1 which were referred 

the Regional Center divided into $2,800, which would have been the cost of the 
service for that length of time on their 1971-72 rate, equals $10.77 per 
request. (We used only the first four months of the year in figuring the 
State B average because their rate of requests more than tripled for the 
month of May, but almost all of these were for packages. Out of 216 requests 
received during May, only 13 were for S ID 1 s and all the rest were for one , 

of the packaged services. The number of requests in State B was lower in 
June and July). State A showed some increase in the volume of requests J 

received in May , but much less than the State B increase, and it is not 
clear that their increase was due to a solicitation of orders for the new 
packages. During June, their requests were back to the former average rate. 

If the c os t - p er - r eque s t is figured for State A on the basis of the first four 
months of 1971 only, it equals $12*25 per request-- the reason it is higher 
is almost exclusively due to the high ir rate of cost of the Regional service 
to State A as reported to us* (This figure comes from dividing a total of 245 
requests for January through April, 1971, into what would have been the 
Regional Center cost at the 1971-72 rate for that time period, $3,000). 
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customers In the state such as universities, not on costs for project 
requests, for which there was no charge). Thus, he urged his project 

staff to send in at least 1,050 requests for the coming yo&v i..i order to 

* 

justify the decision to spend $8,400 for the regional services, and he added 
that given the nature of the packages their project staff might well even 
get 10,000 requests for the year and thereby heavily utilize the regional 
services. His admonition to the field agents was followed by a tremendous 
upsurge in the numbei. of requests received by the reference center and relayed 
to the Regional Center during May- -the volume of 216 requests for that month 
was more than triple the former average for 1971 of 65 requests per month. 

Almost half of that mouth’s requests were for quarterly review packages. Onlv 13 

i 

were for individual computer searches and the remainder were for the other 
two packaged services. ^ 

We; attempted above to rxgure a cost per computer search for the initial year. 

If the State B pattern is followed and the number of requests increases so 
drastically the coming year, then the cost per request will be much lower, 
but this will in great part be a cost per package rather than per individualised 

• . I 

computer search. i 

i 

State G has not as yet been able to arrive at a figure for the 
eost-per-search of their computer service. The project director explained 
that there are so many complexities in assigning total overhead costs to 
project requests, non-project requests (their information center answers a 
significant number of requests from school districts in the State other than 
the two target districts) and the State Department of Education at large 
(since the computer services the whole department) that it has not been 
possible to settle on a single eost-per-search figure. Thus, we do not have 

a comparison of the costs- of providing computerized retrieval capacities 

O 

a State with the use of a more distant source for computer searches. 

77 



In general, to the evaluation team, the experiences of States A and B 
during the initial year o£ the px’ojeet suggests that a State starting an 
information retrieval operation should not expect that dependence on another 
State or on a far distant source will provide a satisfactory computerized 
retrieval capacity; if they opt for such a service, it should probably be 
viewed as an initial stopgap measure. Realistically, they probably should 
assume that development of their own computerized capabilities will be necessary* 
Thus, they can tackle from the outset the issues of obtaining equipment and 
personnel necessary for that capacity and problems of making it operational, 
a pfocess that is likely to require some months of effort. Both western 
States in the project have moved in that direction. If they had envisioned 
that necessity initially, they might have become independently operational 
in a shorter time ana also been able to arrive at more realistieoassessments 
of the cost of a retrieval operation. 

E . Manual search vs computer utilization 

The issue of the quality of material returned to users by an information 

| 

service obviously involves many more factors than just the mechanics of a ■ 

computer operation. Undoubtedly, as retrieval staffs gain experience and ' 

begin to handle larger volumes of requests, they will begin more and more 
to discriminate among requests and to develop considerable differentiation 
in procedures for answering different kinds of requests. For the retrieval 
office the basic question may boil down to a constant balancing of considerations 
about the quality of output against considerations of cost. 

One reference center director recently wrote, for a quarterly progress 
report that ,r £or general topics, manual searches are too costly and time 
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consuming 11 ; while "for highly restricted or specialised search topics , a * 
manual search is the most efficient method to used 1 A specific example cited 
was a. recent request for information on auditory programs; the user was 
concerned not with the teaching of the deaf but of hard-of-hearing pupils * 

This reference center director knows from past experience that there is not 
much in ERIC on that specific subject and that the results of a compu 
search are likely to be disappointing, with much of the output concerning 
deaf children and little pertaining to the rather different problems of 
teaching those who have only impaired hearing. For this particular request s 
, he would decide on a manual search j concentrating especially on doctoral 
dissertations and master f s theses, But such a search will require five to 
seven hours of staff Lime , so few cases per week can be given such treatment* 
That reference center had a staff member who enjoyed manual ^searching , but 
even so, they had to limit the requests so handled to two or three per veak* 
This director thinks that a retrieval center might well have one person on 
its staff assigned to do nothing but manual searches , but this is not possible 
with the size of the staff they now have - 

Thus 5 a retrieval center might anticipate that they will need to make 
decisions about operational procedures along this line --which requests can 
be adequately and most economically answered by computer service and which 
ones should be answered through manual searches , ignoring the computer; and, 
if they are going to offer the latter service, how it will be provided. 

In State C, as mentioned before, this issue is first confronted by 
discriminating among requests and using considerable variation in coding 
procedures for the computer search, depending on the type of request. Initially 
their retrieval staff was doing a hand search of GXJE routinely for each 
request (only ERIC d ata_was on the ir computer), but a time study prompted them 
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to add this resource to the computer data. Now their retrieval staff may. 

check the Education Index if very little is produced by the computer search „ 

/ 

Manual searches of other SEA sources are conducted and additional material 
may be provided through an arrangement with the State library. 

Operational decisions on this issue will J * t-^ly change from time to time* 
As tho volume of requests increases, a retrieval center might become less able 
to offer the kind of service implied by manual searches because of the necessity 1 
of handling all requests faster and more routinely. The solution might 

lie in the enlargement of its own staff to cope with the bigger volume and the 

* 

specialization that this might allow for staff members , or in arrangement a 



with otner institutions which might provide tne service* AXsc^ operational factors 
may be changed by the addition of new sources of d&ia to the computer* or by 
making computer searches for specific topics more economical through modifications 
in the QUERY program or changes in coding techniques ordering the search. And 
the general situation will change as well* Judgements on the issue of manual 



versus computerized retrieval of information, while they will vary according to 
the situation and capacities of each retrieval center, ultimately rely on the 
technology currently available, and this is in a stage of tremendous development. 
Given the best programs now available for computerized retrieval of information 
and the best operation possible of those programs, requests of a highly specific 
nature may as yet be more economically and efficiently serviced through a 

manual search than through a computer search,. But as soon as new programs are 
developed or current ones improved, the point at which that line is drawn will 
have to be reassessed. 



F « he QUERY Program 



Our earlier conclusion that reliance on distant computerized searcher 
-Will prove unsatisfactory does not mean that setting up a local system will be 
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easy sledding. As mentioned previously. State A decided before their fie-ld 
/ ' 

agents started work that the State should purchase the QUERY program and 
develop its own computerized retrieval capability. They expected to be 
operational by December, 1970, although they had several more months after 
that of "free” service from the Regional Center available. But after the QUERY 
tapes arrived and were installed, they discovered in January, 1971, that 
their own cost was $40 per search. At a meeting In March, staff members 
reported to their project director that they had only nine returns out of 
their most recent batch of 12 requests, that the tv '■p — around time on their 
own computer (which, is a part of that State’s computerized system for school 
statistics and data) was still one week, and admitted that "we’re certainly 
far from achieving the depth in coverage that’s coming out of the Regional 
Center 1 , One staff member explained her opposition to u s ing ' Lb c ir QUERY 
package "as it exists now” in economic terms; "we’ve already spent $975 
thia month , Li cost us $600 to run 15 searches ... The current package ties 
up the entire computer." 

The project staff, working with personnel of the State Board of education 
computer center, made continual efforts to solve these problems, and by 
the end of April the cost had been reduced to $15 per search. They aimed 
to have it further reduced to $7 per search by the summer, but only a slight 
reduction-- to $14 per search-had been achieved by September, However, 
their own turn-around time for individualized searches was only one week. 

State A discovered — as, apparently, do most QUERY users— that a great deal 
of time and expertise is required to make the program operational and efficient; 
modifications to fit each individual installation are likely to be necessary. 
State B, in the spring of 1971, also purchased QUERY and tapes with the 
ERIC data for a computer in their State, The program was installed in June, 
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but is not working as yet. A consultant from the contractor hold a two-day 
tiainmg session then, hut the retrieval center director indicated that more 
training, even on coding techniques, will be necessary. With the experiences 
of the other two pilot States as a warning, the State B staff may have realistic 
expectations of the initial difficulties likely to be encountered with 
QUERY. In reporting that the newly installed program was not yet working, 
the retrieval director said, "But we have a year to do it," referring to their 
prior purchase of the regional service for the coming year. 

State C had obtained the computer search capability for ERIC, or the 

> ■ 

QUERY program, before their project began. A history of their experiences 
is probably typical, 

* V ' ^ ’ m '* rujeis of the retrieval staff had to learn how to use QUERY, 
or how to code requests for the computer. The initial training session by 
a consultant fiW the QUERY contractor was judged very good by the two members 
of the reference center who were attempting to gain this new skill. A few 
weeks later, at the time of the first site visit of the training team, computer 
•' Searches were bei “S successfully completed. But searches were taking four 

five hours of computer time, according to the new director of that State's 
computer center. The State department of education had installed a new, 
faster computer over the summer. The computer center director estimated 
that on their former, slower machine, searches under QUERY would have taken 
10 to 12 hours of computer time and that this would have been impossible. 

Searches on the new machine were possible only because the capacity of the 
machine was not being fully used as yet and it was a multi-job machine 
(handling up to seven, different jobs simultaneously). 
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A member of the training team for the pilot states discussed coding 
procedures and their specific problems with State C staff members on the 
first site visit of the training team. This appeared to be an exeeddingly 
helpful session* as reported by a member of the evaluation team. One of 
the retrieval staff members recently recalled that the discussion may have 
been somewhat useful but that the main problem facing her and her colleague 
at that point vac Inch of experience. They understood theoretically how to 
do the coding; what they needed was more practice. Experience did increase 
the coding efficiency £o the operation to some degree; however* they were 
able to code only a four to six searches in a batch which required a minimum 
of two and one-half hours of computer time. This still was far from satis- 
factory. After the communication specialists were named for the State f s 
target districts and began their work* the number of requests for information 
began to increase significantly. With the higher volume* the information 
center could not keep up with the demand. Also, even though the staff 
members became more expert* the laboriousness of the coding procedure was 
a continuing problem and a serious obstacle to the whole operation of their 
service. QUERY was operational* but it wasn’t efficient. 

One of the staff members in State C recently observed that the critical 
fault with QUERY is that the program* as written* was set up so that a request 
could be very specific about the topic of the search* but that also, unfortunately* 
the request had to be very specific. Under QUERY* their limitation for an 
order to the computer is set in terms of the number of lines of instructions* 
they can send a maximum of 30 lines of instructions (or coding) for each run 
through the computer (or batch of requests) * regardless of how many individual 
requests (or searches) may be included in that batch. But at that time* each 
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individual request required eight. to fifteen lines of coding, which meant 
that only a few request could be handled simultaneously within the 30-line 
limitation. It took a retrieval staff coder about a half hour for each case, 
in addition to severely curtailing the number of cases which could be in- 
cluded for each computer run. The result was four to six requests included 
in a batch, at a maximum, and each batch required two and one-half hours of 
computer time as a minimum. Also, a detailed time and motion study of the 
operations of their office in January indicated that all the steps in the 
processing of eaph single request required approximately, three hours of staff 
time* 

i 

The State C staff felt that changes for greater efficiency were necezsarv * 
Coincidentally, several members of the staff heard a presentation made by 

K*' 

Dave Altus, associated with the ERIC Clearinghouse on Rural Education and 
Small Schools in New Mexico, while attending an educational conference* He 
described modifications he had, made in QUERY for his own use, and subsequently , 
State C asked him to come as a consultant to help them with their own QUERY 
problems. Altus came in Janinry, and as a result of his modifications in 
the QUERY tape and in coding procedures. State C was suddenly able to process 
up to 25 requests per batch in 15 minutes eompu er time. Coding a single 
request now takes about five minutes (compared with fifteen minutes previously) 
and staff members find they can code a batch of 15 requests in an hour and 
a half. Each request now requires only one or two lines of coded instructions 



generally; thus, many mere requests can be included in the 30-line limit 
per batch 

Could the State C project have been saved months of problems and much 
cost if they had been steered to Altus before they accidentally heard his 
speech at an educational conference? Possibly; but also perhaps not, 
according to a retrieval staff member: "It took us that long to know the 
changes we wanted to make in the coding* 11 One would hope that not everyone 
who begins his experience with QUERY in the future must necessarily follow 
the same circuitous route to making the program serve him efficiently, but 
that some of the difficulties can he avoided on the basis of the previous 
©y erience of other users. 

Other changes introduced by State C in its operations included obtaining 
the computer tapes of CTJE (Current Index to Journals in Education) on which 
they had been previously doing a manual search for every request. The 
reference center has now eliminated almost all the manual searching of CIJE 
done by their staff, cutting out approximately one and a half hours of the 
staff time formerly required in the processing of a request, (As mentioned 



* 

In brief, the coding changes introduced by Altus made it possible to code 
only parts of some words involved in the search topic rather than every letter 
in the word (e *g, , "dary instead of "secondary M ) , and allowed coding of only 
non-asterisk ones* (In the coding procedure, asterisk terms are the primary 
subjects or topics of the request; non-asterisk ones are the subsidiary, less 
important or more general ones. Previously, the coder frequently had to 
include the same word preceded by an asterisk and again without an asterisk*) 
In addition, coding instructions now specify which ERIC clearinghouse data 
the computer is to search* Data on the ERIC tape is stored according to 
the clearinghouse from which it originates, and- -although there are some 
ambiguities or overlaps- the clearinghouses are distinguished by the subject 
of the materials Thus, one can specify the single clearinghouse, or possibly 
the two or three, which would have research on any particular subject* This 
change accounts for the big decrease in computer time required per batch, 
since the computer no longer searches the entirety of its data bank for 
every search , , 
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elsewhere 5 the state library does manual searching for them.) A 1 so, as the 
volume of requests increased- - to slightly more than 100 in January and again 
in June , with levels generally near 70-80 per month in the intervening 
period-- the staff of their information center was enlarged ■ * 

Another improvement in their computer performance has been achieved 
very recently » After installation of GXJE, the computer performance has 
been achieved very recently* After installation of CIJE, the computer was 
not printing out the journal volume or page number. Staff members had the 
time-consuming and irritating task of looking up all those numbers in the 
CIJE printed index and copy ring by hand onto the printout intended for 
the requester the journal cit&ions for each document * They assumed that 
this deficiency was built into the program, but on a trip to the ERIC 
information service in another State, one staff member noticed that CIJE 
titles and authors were supplied by the computer on printouts » Thus, the 
State C crew realised, as a result of another fortunate coincidental encounter, 
that the deficiency was either on their own tape or in the way it was operating* 
State G staff members began making phone calls-- to Washington, to the retrieval 
staff in another pilot project, to the original consultant for the QUERY 
contractor, and to their onw computer center director. Finally, they were 
able to determine that an instruction card was missing for the CIJE printout 
program* Again, because of a relatively accidental observation and considerable 
initiative in trying to find an answer, several minutes of staff time have 
been saved on every request. A new time study done in June showed that, 
as a result of all these changes, the staff time required for servicing a 

-k — * *" — 

State C now has eight members on its retrieval staff: the director of the 

unit, 7 ho is different from the director of the project, who is in the state 
department of education; four professionals; and two secretaries and clerks * 
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request had been cut in half since January, or donw to 90 minutes per request* 

II • Furnishing the Client With Complete Copy 

Another issue confronting all three States in the project wag how to 
provide complete copy of reports or articles in which a requester decided 
he was interested after seeing abstracts of them. The various questions 
which had to be investigated and answered by each State included the following: 
A . Microfiche copy or hard (p r inted ) copy 

While there are obvious advantages to providing a requester with a 
printed copy of an article in which lie has indicated interest, technology 
has/ yet to make this economically feasible in all cases. In general, the 
States have decided to return microfiche copy. Articles or references 
resulting from manual searches, as opposed to the computer search, and various 
other materials and ^oources may be supplied by xeioxing or other me Lhuus 
of copying. The PREP packages, for example, are duplicated in bulk by 
two pilot States and made widely available through the information center and 
field agents. But complete copy of ERIC items is usually provided in micro- 
fiche form because of the high cost of reproducing printed copy, and the 
field agents have then the additional role of acquainting requesters with 
new equipment-- the microfiche reader — and providing it for their use. 

In. State A, the initial return package from the computer search includes 
a cover sheet listing all the various places in the State, such as State 
colleges, etc,, where either microfiche copy or hard copy for the articles 
cited on the printout may be obtained. The retrieval center does not provide 
complete copy to clients as a general rule, although field agents in the two 
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target districts may get microfiche copy for tlieir requesters. Also* the, 
retrieval center director reported , if a client call n and says there is no 
copying facility available or convenient for him to use* the retrieval 



center would make arrangements to get microfiche copy for him. The retrieval 
center does not have the machine for producing microfiche copies* but the 
State library has the microfiche collection and ill send out microfiche 
copy and portable microfiche readers on loan. The retrieval center does 
have a machine for making bar. I copy from microfiche and supplies this seivlee 
for personnel in the SEA on requests. 



Nor does thfe central office of the State C project have the capacity 
to duplicate microfiche. They generally order microfiche copy from an 

i 

adjoining State — in £«aut, this is the one service for which they still rely 
on their neighbor Stare* (In the original proposal* they intended to depend 
on the other Stated information service for ail of their computerized 
retrieval capacity.) However* available resources for copying within their 
own State are utilized in various ways. The field agent in one target district 
has the facilities of a microfiche reacbr- palter and the microfiche collections 
in his district, A State college for women is in the other target district, 
and the field agent there can get hard copy through them and utilize their 
microfiche collection. This field agent scans the complete copy of the 
microfiche document and makes hard copy of a few pages which she regards as 
especially useful. She then returns these pages with the whole article in 
microfiche to the requester and then explains that other pages can also 



be copied. She feels that this encourages the requester to tackle the 
microfiche reading: the printed pages may pique his interest and suggest 

that he will not need to take notes. He can simply check specific pages he 
wants for future reference* and thereby confront the microfiche reader only 
once for a given article. 
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B • Pa y lng for c cm p lete c opy 



The State A staff phrased this issue ? on the agenda of one of the 



^initial staff meetings 5 as n Fee or free to client ?’ 1 Since the client In 
non-target districts makes his own ai’rangements to get complete copy from the 
institution nearest him with cop3^ing facilities, he pays for it himself « In 
the target districts* the project- furnishes microfiche copy to clients 
through the field agents* one using the facilities of the State library and 
the other those of the State university* 

In State C 3 eorp ' 1 ere copy- ~whe tlier in microfiche, form or in hard copy*-- 
* 

is provided free to clients in the two target districts* Requesters from 
other school districts in the State must pay for their copy* In State B* 
the project might pay for one or two pages of hard copy sent to a requester 



but not for copies of complete articles* They intend next year to send out 
microfiche copies on loan* In general* clients will pay for copies of articles 
that they request * although this cost may be covered by the intermediate 
agency* or regional office, out of which the field agents operate. Whatever 
the answer* or variety of answers* to this question by a State information 
service * the cost, is nominal for documents in microfiche form but can mount 
to several dollars per article for hard copy* 

Cm . Microfiche hardware '' 

In the preceding discussion* there have been repeated references to the 
basic microfiche data collection* the microfiche reproducer* the microfiche 
reader-printer (for producing hard copy) and microfiche readers* It is clear that 
the availability of these resources will be essential to the functioning of a 
retrieval office. And although "hardware" might seem to be a mundane issue* a 
cut -and -dried problem with obvious answers, difficulties during the project have 



aused repeated instances of irritation, frustration or even complete interruption 




28 



of the whole process of supplying information to users. One of our observers in the 
early stages of the project found a retrieval staff director completely disgusted 



after his first attempt to provide hard copy of an article with the available micro- 
fiche reader-printer. Various field agents have voiced dissatisfaction with certain 
microfiche readers, or reported that sine© their single microfiche reader was not 
working and no others were available in the area, none of their clients could read 
copies of articles, Jn general, each state has had to devote considerable time and 
effort during the year to determining the availability of equipment f how much ad- 

A 

ditional equipment would be needed for effective functioning, and whether certain 
models or brands are preferable, 

D * One Sta t e ' s experiences 

Documentation of difficulties caused by these issues is niore frilly 
available for Shat© R- so experiences of that State might be seen as a kind of 
case history. Tills pilot State had not made decisions on these various points 
until the project was well underway and the field agents had been at work for 
some x?eeks . They may have assumed that hard copy of needed articles would be 
provided by one of the three universities in the State which had the ERIC 
microfiche collection. In mid-October, they discovered that the cost of this 
wou) d be prohibitive: the rates charged by the three institutions ranged from 

10 to 25 cents per page. Even if there were a slight discount for the project, 
their budget could not absorb such charges, and few requesters were likely to 
be willing to pay several dollars for a short article, much less a conceivable 
$25 for a long government report. State B had on hand at this time a dozen 
requests for complete copy of articles or reports abstracted in the profiles 
returned to clients. The retrieval center director determined that one of 
the requested reports had been pub 1 i sh c d — bu t the cost was $19. On the other 
hand, the State department of education did not have the microfiche collection 




of ERIC and ^IJE or the capacity to copy microf iche , and the schools in 

i 

general .did not have uiicrof iche readers, The tnito St^tci universities 

which had this would riot circulate their collection, nor could they copy 
it in microfiche. Hard copy could be ordered from the national supplier , 
but this would take a great deal of time for er~h request and the cost 
was only slightly lower than the in- State reproduction cost. Thus* State 
B was caught with a backlog of requests for complete copy and a dilemma 
about how to service them, since these basic questions had not been 

decided in advance. Although decisions on these issues were finally made, 

* 

State B field agents had difficulty getting complete copy of an article 
or report for a requester throughout the year and were able to return 
complete copy, at least of EPvIC or CIJE materials, in relatively few 
cases 5 until the summer of 1971--some nine or ten months ’ after the project 
had begun and six months after operational decisions had been made. 

The combination of dissatisfaction with the Regional Center service 
and problems in delivering copies of complete reports or articles 
(either in hard copy or microfiche) led State B to consider various 
alternative plans of operation. At a staff meeting December 2, 1970, 
these alternatives were discussed. In summary, the deicisions made then 
were the following: 

1) The Field Agents should have copies (in printed form) of various 
indexes of educational research: specifically the fill (for ERIC), CIJE and 
Pacesetter , At present, field agents also have PREP kits and will have NCEC 
materials as they become available , The plan was that the field agents (or their 
secretaries) would do a manual search as soon as a request for information was 



received, and that by xeroxing copies of the abstracts listed in source close at 



hand would be able to, return some pertinent abstracts to their requesters within 



a very short time. 
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2) The state department of education, or the o\:a1:c information center of 
the project, would purchase the complete microfiche collection of ERIC end CUP; 4 

I 

Thus, they would bo able to supply complete copy of documents from choir own resources 

3) Field agents will have microfiche readers that could be loaned to 
requesters who wished to read complete copies of those articles in which they were 
interested. 

A fourth decision, finalized later, to pureh^e the QUERY program and 

i 

computer tapes of ERIC and CUE data, has already been mentioned* 

But once these decisions were made, theStatsP team faced what they felt to 
be endless frustrations in putting them into effect. Long negotiations were 
necessary to work out arrangements for the assignment of a computer suitable for 
QUERY to the state department of education* Although resources and equipment 
needed for the new plan of operation were ordered, there were delays in receiving 
almost every part of it* Even obtaining the printed Indexes for the field agents 
took some months* The purchase of the whole microfiche collection for the state 
department of education, by state law, had to be let out for bids, even though# 
it was available only from one source in the nation* Since the national contractor 
was Changed at about this time, the original purchase order was returned, and the 
whole bid-letting procedure had to be gone through a second time before a new 
order could be sent in. 

Although most of these problems had been cleared up by the summer , 1971, 
there may be further complications in the manner in which the new plans for 
operation actually work during the coming year* Even though the field agents 
have their basic resources and may look up descriptors, they have not actually 
done real searches* Thorough individual searches on a topic are still supplied 
only by the Regional Center with its lengthy turn-around time* Hopefully, 
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this will be remedied when QUERY is made operational in that State. In 
practice, the field agents have been relying more and more on packaged- 
services, looking up what appear to be relevant categories in the listings 
of these services supplied by the Regional Center. Insofar as supplying 



complete copy is concerned, the retrieval office plans to loan the microfiche 
copy of any requested document from- its newly acquired collection. But 
they do not have the capacity to reproduce microfiche, nor does their SEA, 



so they are constrained to send out their only copy. If they get numerous 

requests for the same, article, the} 7- will make arrangements with a local 

commercial establishment to make extra copies of that document , but they do 

not plan to duplicate nhe requested microfiche copy routinely. Finally, 

field agents have only one microfiche reader apiece, which can be a major 

k 

problem in States with target districts which encompass a large geographical 
area. If the reader is not working or is out on : loan to another user, then 
a requester may have to wait a considerable time after his microfiche article 
has been returned before reading it* The field agents might take, the machine 
' with them or keep it in their office, rather than loaning it, but it is not 
feasible for a client to read a long report or document while the field agent 
is waiting around. On the other hand, if he leaves the machine with the 
requester, some days or even weeks may pass before he can pick it up. The 
distances traveled by field agents servicing multiple school districts spread 
over a large area create problems which would not be as serious for an agent 
servicing a single school district or town, or even a county. 
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Ill, Record Keeping and Filing 

A third major area to which retrieval staffs have had to devote 

I 

considerable attention has been that of filing and record keeping. Vnr the 
most part they felt it was accessary to determine through their own 
experiences what systems best serve the needs of the project in their state. 

At various times during the year, each state has some modifications in 

their filing and record-keeping systems. Again, this may seem a mundane 
issue worthy of little mental energy, but experiences of the pilot states seem 
to indicate that it is worth considerable effort- -and even more usefully, 
considerable forethought and analysis-~beeause of the effects that the systems 
being used have on efficiency of operation. In several instances retrieval 
personrnel in the pilot states have reported how worthwhile it was t n add 
new files or change the way of keeping a former set of files because of 
the surprising amount of time they could save. 

One member o:t the training team for the pilot: states, who has con- 
cantrated on these problems, has been, praised by retrieval personnnel. The 
sessions that he conducted at the lirst training workshop, his consultations 
with individuals and suggestions about problems that arose, as well as 
examples of systems used by a retrieval center that he directs^-all have been 
cited as most useful. 

Although each state devised its own systems and procedures initially, 
we have noted that the different state retrieval offices have opted frequently 
for the same or a very similar solution to a particular problem. This may 
result from the training team * s influence, or from communication among the 
pilot states --or it may be, as we suspect, that there is one answer which 
has definite advantages over others. 
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The basic element that a retrieval office deals with is an individual 
request for information. Thus, the primary record-keeping system should be 
in terms of a specific request for information, and not in terms of the 

V 

requester, (The obverse would be true of a doctor's office, where the basic 
unit to be serviced is patients, or the whole person and his history, not 
the current complaint). Record-keeping difficulties seems to result if a 
particular request z.z primarily classified according to the person making 
the request, nr- th» <a..h -ject on which information be available or 
requested, or the nature of the service requested— since persons may make 
multiple requests, a request may pertain to multiple subjects, and a single 
case may be serviced in multiple ways. Thus, the best system seems to be the 
assignment of an identification number to each request, numbering ’’he 
requests chronologically as they are received by the reference center. 

(The training team has suggested preceding these numbers with the last two 
digits of the year— e.g., 70—0001, 70—0002, etc., then 71—0182, 71-0183, 
etc.). Other matters, such as • keeping track of the topics of requests or 
of the requests from each field agent, can be handled by simply cross- 
referencing in subject and person files. 

9 

In State A when a new request is received, it is placed in a manila 
folder these have nil been pro -numbered chronologically -- and a complete ease 

record is started. Every record pertaining to the case and the servicing of it, 
including a list of all bibliographies and materials returned, goes into the 
folder. When the request has been answered and work on it is finished, the 
folder is moved from the "Open" file drawer to the "Closed" file drawer. In 
addition, the office keeps a rotary subject file : all the informational resources 
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and materials in the office have boon numbered- on trios under any partieuj -r 

I 

subject in the rotary file include this number which , in essence, tolls exactly 
where in the office the material may he located. The office also has a box of 
index cards, filed by subject, of all the searches they have done, previously. 



State b initially decided to use the request fo^m issued by the Regional 
Center, Some difficulties resulted when these were all prenumbered 
and different hatches given to different field agents* The field agents did not 
necessarily use tlieir forms in the numbered order ana rorms were being 
returned simultaneously by all agents, each of whom had numbers in a different 
range . Thus, there was no numerical order of the identification numbers received 



by tho inference center* 



In addition, no number at all was assigned to a request 



that was not to be forwarded to the Regional Center tor a computerised 
search. They have now devised their own information request form and these 



will be chronologically numbered. 

State C gave considerable thought before their project began to 

the forms they would use for their records and the different files they would 
keep* They also kept a case record file (chronologically numbered) that detailed 
the processing and servicing of each request. But there was one page, the 
bibliography sheet listing all sources cited to tho client and materials 
returned # tbat was put into a subject file together with a copy of the computer 
print-out. ?x± the spring, 1971, they began to recognize that some requests 

were almost exactly the same as previous ones. At one time the newsletter of the 
school district served by one field agent listed all topics on which 



she had provided information to date. The state office was thereupon flooded 
with 60 requests for duplicates of one or more of the previous searches. Pulling 



out the materials from the various parts of they subject file putting- the .cases 
back togotho r 9 so to speak was enormously time consuming s and the 

i 

staff decided there would he greater benefits from keeping a file of each complete 
case- They instituted this new procedure for requests and, gradually, as staff 
members found time, they re-as r? ombled the files for cases handled earlier durj ng 

the year. Staff members cited the change with great enthusiasm, and long before 
the char, go "-over job wau completed, were already finding "a tremendous time saving” 
under the new method , 

State C aloe keeps a copy of the actual coding instructions for 
the computer search in its case record. The old subject file will bo maintained 
but will house resources other than case records that is, all the materials 
on various subjects that come into the office and may be useful on a requesr e 
They also recently developed another file in which a card is made for every 
descriptor- in the ERIC Thesaurus implied by a request. These cards, with the 
exact wording of the request and its case number, will be filed according to 
descriptors. 

These observations should suffice to indicate the direct consequences of 

j 

office systems on the efficiency of an information and retrieval service. An 
additional reward of effective and serviceable record-keeping systems may come 
in demonstrating its worth to local and national supporters* Most such centers, .■ 
if set up as a new service of the state department of education or on an experiment 
al basis, will at some point or another have to justify their existence or, If 
supported by federal funds, to persuade the state to foot the bill when that 
source of support is discontinued. Numbers alone may be insignificant p but an 
effective record-keeping system should provide materials that will greatly 
facilitate the job of summing up results claimed by the service, 
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those in futx'J'o projects 



Aa the history of the pilot states implies » 



\VilO til?© ut hQ 

to visit an 
keeping and 



opting to establish a nov; information retrieval system would do well 
existing office and analyse in great detail their systems of locorcl 
filing. In this area, there is now a supply of expertise and 



experience which should be drawn upon in advance. There ca* be little advantage 
in developing one f s own systems through trial and error. Indeed, harsher terms 



might rightly be used: for future states there should be no excuse for the 

time and effort wasted by necessary redoing and remaking of systems once a project 
is underway. Modifications to meet idiosyncratic needs of a particular state 
can be added without unseating basically satisfactory systems. Primary roquiremen 

i 

are sufficiently general isablo so that the systems ran bo organized in ^civwicg 

And they should be established from the outset, even though they might seam 

»*» 

overly elaborate to the neophyte. 

IV. Staffing 

Summarising the ideal background and qualifications to be sought in a 
retrieval staff member is impossible for reasons which should by now bo obvious * 

j 

the diverse tasks that confront a retrieval system and reference center imply 

i 

requirements of an equally diverse range of background skills and experience* 

A partial list of the capabilities that a retrieval staff should have at the out- 
set or attain in short order includes the following: 

1) computer skills; 

2) library and research skills, or experience as a research librarian, 
most usefully in an education library; 

3} familiarity with educational hardware and technology, and 
preferably experience in judging and procuring such equipment; 
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) <. -porter « i n of f i?ys Sort's rvA wanjjfjeipcrvfc , record anri 

filing, 

l 

This excludes a matter not dealt v/ith in this paper but which some would 
considoi^ the prim© characteristic needed for retrieval staff members; considerable 
knowledge of educational research (knowledge of its substance, not just how to 
obtain it); familiarity with the current status and directions of research and wit 
the institutions who f,h« most innovative new practices are ©merging: and 

finally , and most important , the perspectives, standards and experience for 
evaluating the wo;.d:Tv of research data, new proposals and experiments. An 
ac q n a i n t an o © with the state * s schools, their character! stj x*s and personnel, 

might well prove an asset which would be invaluable. 

Rarely are those qualifications found in one person* Future states might 

A 

wish to anticipate that they start with a small i©ti*ieval of flea and staff, 

as did one of the pilot states c But they should realize that as the volume of 
requests increases and as the scope of the office enlarges to take on all these 

diverse functions of an information service, a multi-person and multi-skilled 
staff will almost certainly be a necessity. 

A couple of other points might bo mad©. Although computerized retrieval 
of information and research was a basic part of the project, the directors in the 
three pilot states apparently did not make computer experience or expertise a 
requirement for retrieval staffs. In all three states, only one person employed 

for the retrieval operations had any prior experience with computers. 

There may well be valid reasons for the Director ! s decisions not to look 

for such competence , but it should be emphasized that even though the retrieval 
staff may look elsewhere for actual computer expertise, they will surely need 
some knowledge, at least on the theoretic level, of the possibilities and limits- 
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. tiona of computerized retrieval of information, Even, though it, me.-,- not be noco 
for the center to have a computer programmer, some staff members will have to 
learn at least the coding techniques for the specific program used for the 



computer searches# 

■ Another point is that' some of these skills and experiences may be <■ r 
found more readily in fields other than education. Even though the retrieval- 
information office is set up in the state department of education, future state 

directors might look for some staff members outside of educational circles. 

* 

Such diversity is more feasible, of course, if the staff of the retrieval 
center is larger. Indeed, the information center with the largest staff 
among the pilot states does have some members whose degrees were not in 
education. 



The benefits from the division of labor according to the talents and 
propensities of individual staff members have been cited by two states. (The 
other pilot state has only one person with part-time secretarial help on its 
> retrieval staff). In State A, one staff member, who majored in English 
education enjoys doing research and individualized manual searches, while 



another, whose background was as a school administrator, concentrates more 
on other aspects of the service. State C, which has four subordinate 
professionals under the center director on its staff, reports a fortuitous 
meshing of talents. One English major especially likes writing jobs, while 
another staff member prefers doing only the research, gathering all the 
relevant data on a subject. As a writer-researcher team, these two have 
already prepared one long report, the first time that state has attempted 
one of its special aims: not only gathering all available information on 



) urrently 




important 



educational 



topic in 




the state but also correlating it. 
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synthesizing it* ' v.Ticing j t lu j in a paper » a ill 5 «r cl m ember of t-hc 

office p with 3- social science major in college, has taken. special responsibilities 
in overseeing and maintaining the office’s record-keeping systems. The fourth, 
formerly a teacher who specialized in reading, is the only one of the staff 
who has been with the project since its inception and who helped write the 
original proposal. She was also one of the original pair who learned QUERY, 
and knows more about the field of education and educational developments than 
others on the retrieval staff. In short, diversification of background and 
interests seems highly uenefieiai, 

V. Rec oimuendati o n g Tor Future Projects 

A summary statement of some of the recommendations suggested by the ex^ 
periencos of the first three pilot states might he useful f$r futm e states 
attempting to establish information dissemination projects . 

1) A state should plan initially to provide its own computerized search 
capacity. Reliance on a regional installation means that the information service 
will have little chance of influencing such matters as turnaround time, relevance 
or adequacy of the computerized search. Thus, resorting to a distant computer 
service should he viewed at best as a stopgap measure, one that will ultimately 
prove unsatisfactory. 

2) A state should anticipate difficulty, frustration and delay, pro' ably 
more than the contractors would predict , in making the QUERY program for com- 
puterized searches operational and efficient in their own installations. This has 
happened so generally that an information service making the decision to install 
QUERY might well plan to send their computer and retrieval personnel to an in- 
stallation with a similar computer capacity and set-up which has already been 
through the process, to study in detail the* pitfalls and obstacles which prior 
projects have already experienced and found ways of overcoming. 
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) fp} je office of Eg ac o •: 1 on or td'e QUERY contractor should alia at nnx**e 
maiding new installations attempting to achieve computerized search capabilities 
through the first few months. Their responsibility should extend to the point 
where the program is operational — and, to some extent, efficiently and 
economically operational — not just to the point of purchase and installation. 
Perhaps detailed, step-by-step case studies of the experiences of those who have 
installed QUERY could he compiled and furnished to newcomers to the process. Or 
perhaps the Office of Education or the QUERY contractor should provide task force 
teams, or more cqnsul-— .is, or more extensive and specific guidelines or analysis 

of the individual problems facing a new installation. Whatever the method, our 
impression is that the pilot states have not had enough expert assistance on this 
score. New state projects should he able to purchase that expertise and service 

when they pur she so the program. 

4) The rationale behind packaged information services should be well 
understood. To the extent that they are used, it should he with full awareness 
Of possible differences — in approach to users and in their likelihood of 
effecting change — between information supplied on this basis and the aim of 
individualized service. One must not simply assume that the appeal of packaged 
information to clients lies In its content. For packages relieve school personnel 
from the difficult task of defining exactly what their own need or problem is, and 
thus , eventually from the burden of evaluating the applicability of specific in- 
formation to their own situation. This is especially true of the packages that 
are intended simply to raise awareness of certain educational developments. 

These packages require nrobing ^low-ups to see if more detailed information or 
consultation is desired. However, if packages are disseminated broadside, 
individual follow-up will become virtually impossible. 
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j) j oviLi. cux vices si'Olft the outset Should adopt; a stance of’ continual 

reassessment of their modus operand! , A constant balancing- of cost factors 
versus the quality of the output and service, of what is feasible with available 
staff and what is the potential of available technology — these arc the basic 
issues behind policies about whether requests will be- answered by manual searches 



01 by computerized searches; and which should be serviced by packages and which by 
individual searches , 

S) The computerized search for abstract* of existing educational research 
is only the first step toward the real aim of the information service, which is 



to pro” > de complete copy of research documents, articles or reports (or sections 
therec* , which would be useful to a client. Procedures for completing the process 
should be determined iii_ advance : In what form should complete copy be provided 
microfiche or hard copy? How and where can either microfiche or hard copy be 
obtained/ If microfiche is the only feasible format , what techniques can be 
adopted to overcome the obstacle this presents for users? How much will complete 
copy cost and who will pay for it? What hardware is available and what will be 
necessary for making the whole process function? How can the necessary resources 
and hardware be obtained? 

7 ) Personnel establishing an information service would do well to visit 
ah existing retrieval service and analyze in detail their record-keeping and 
filing systems. Basic systems should be outlined in advance and maintained from 
the outset. Considerable expertise and guidelines on this score have evolved 
through the trial-and-error experiences of existing retrieval services. 
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8) The? entire range of qualifications trad capabilities required Ly an 
information service staff should be envisioned in advance, and the staff should 
consist of more than one or two individuals. Thus, a project director might 
consider which characteristics are most essential at the outset, how certain 
necessary capabilities (perhaps-, for excimplc, computer expertise or familiarity 
with the potentials and problems of computerized retrieval) can bs provided for 
the service by outsiders ■ r ' not by the initial staff, and the advantages that will 

result from a diversification of backgrounds and qualifications of staff members, 

* 

not only in the initial staff but through additions in the future . 
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